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ABSTRACT 


The  basic  descent  algorithms  for  minimizing  nonlinear 
objective  functions  will  generally  find  a  local  minimum. 
For  problems  with  multimodal  objective  functions,  it  is 
desirable  to  extend  the  search  in  an  attempt  to  find  a 
global  minimum.   Several  versions  of  a  new  method  for  doing 
this  are  presented.   Computational  tests  are  performed  to 
compare  these  methods  with  each  other  and  with  existing 
methods. 


PronarpH  hv: 


A  New  Method  for  Global 
Optimization 
I.   Introduction 

The  basic  descent  algorithms  for  minimizing  nonlinear  objective 
functions  will  generally  find  a  local  minimum.   For  problems  with 
multimodal  objective  functions,  it  is  desirable  to  extend  the  search  in 
an  attempt  to  find  a  global  minimum.   Some  approaches  to  this  formidable 
problem  have  been  surveyed  by  McCormick  [1], 

A  simple  strategy  for  avoiding  local  solutions  to  multimodal 
problems  which  has  been  suggested  frequently  is  to  repeat  the  local 
optimization  process  several  times  starting  from  different  initial 
points  x   ,  k=l,2,...   .   If  all  of  the  optimizations  converge  to 
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the  same  solution  x    ,  then  the  problem  is  probably  unimodal,  and  we 
have  more  confidence  in  the  solution  x   than  if  only  one  minimization 
had  been  performed.   If  several  different  local  solutions  are  found  by 
the  repeated  minimizations,  then  we  are  warned  that  the  problem  is 
multimodal,  and  the  chance  that  the  global  solution  has  been  found  is 
greater  than  if  only  one  minimization  had  been  performed. 

Some  strategies  for  choosing  the  starting  points  for  the  successive 
optimizations  to  improve  the  chances  of  finding  the  global  solution  were 
described  and  compared  by  Hartman  [2].   In  this  paper  a  new  method  which 
emphasizes  the  intelligent  selection  of  starting  points  is  developed 
and  evaluated  by  computational  tests. 


2. 

II.   The  Problem 

We  concentrate  on  the  "essentially  unconstrained"  problem 

minimize   f(x) 

subject  to  x  e  S  C  En  (1) 

where   S   is  a  bounded  subset  of  E   and  where   f(x)   is  a  nonlinear 

function  which  attains  its  (possibly  numerous)  local  minima  in  the 

interior  of  S.   Thus,  the  set  S  serves  only  as  a  bounded  region  within  which 

starting  points  for  successive  unconstrained  minimizations  are  chosen.   A 

convenient  choice  for  S   is  the  rectangular  region 

S  =  {  xeEn  |  L,  <;  x .  s  U   ,  j=l,...,n  }  (2) 

determined  by  imposing  upper  and  lower  bounds  on  each  of  the  variables 
x . .   The  algorithm  of  this  paper  will  be  developed  for  rectangular  S; 
the  basic  idea  of  the  method  could  be  extended  to  more  complex  sets  S 
at  the  cost  of  additional  bookkeeping  in  the  algorithm.   We  also  assume 
throughout  that  the  function  f(x)   satisfies  any  continuity  and 
differentiability  assumptions  required  by  the  unconstrained  minimization 
method  to  be  used. 


3. 
III.   Development  of  the  Method 

Probably  the  simplest  method  for  selecting  the  initial  point 
x  eS   for  the  k    minimization  (k«l,2,...)  is  to  choose  x   ran- 
domly in  S.   A  random  choice  is  not  very  efficient,  however,  since  it 
ignores  all  the  information  which  has  been  gathered  in  the  k-1  minimiza- 
tions already  completed.   The  method  of  this  paper  is  based  on  the  idea 
that  if  the  starting  points  for  successive  minimizations  are  chosen  to 
avoid  re-searching  territory  which  has  already  been  searched  during 
a  previous  minimization,  then  a  more  efficient  procedure  will  result. 

Implementation  of  this  idea  will  involve  the  following  four  steps: 

1.  Develop  a  mathematical  description  of  "the  territory  which  has 

been  searched". 

k  th 

2.  Select  the  starting  point  x  eS   for  the  k    minimization  to 

be  "as  far  as  possible"  from  the  territory  which  has  been  searched  by 
the  first  k-1  minimizations.   Thus  we  attempt  to  ensure  that  each 
successive  minimization  covers  some  new  territory. 

3.  As  the  k    minimization  proceeds,  modify  the  description  to 
include  the  new  territory  being  searched. 

4.  Terminate  the  k    minimization  prior  to  convergence  if  it 
enters  territory  which  has  already  been  searched.   Thus  we  avoid 
repeatedly  searching  to  the  same  local  minimum. 

The  details  of  these  four  steps  will  determine  a  particular  search 
algorithm.   In  this  paper  we  will  describe  three  algorithms  Al,  A2,  and 
A3  based  on  the  above  outline,  and  evaluate  them  by  computational 
experiments. 

Step  1.   The  "territory  which  has  been  searched"  is  described  by  partitioning 
the  set   S   into  disjoint  cells  S±    ,  i=l,...,N.   Then  a  list  of  the  cells 


4. 
which  have  been  "searched"  (see  Step  3)  describes  the  subset  of  S 
which  we  call  "the  territory  which  has  been  searched." 

We  assume  that  the  cells  are  such  that  at  most  one  local  minimum 
is  contained  in  any  one  cell  and  that  if  a  cell  has  been  searched  and 
a  local  minimum  not  found,  then  there  is  no  minimum  in  that  cell.   For 
all  the  tests  described  in  this  paper,  we  have  restricted  our  attention 
to  rectangular  regions   S  as  in  (2).   For  such  regions  an  easily 
described  partition  is  given  by  partitioning  the  j    coordinate 
interval   [L,,U.]   into  m.   equal  subintervals   (j=l,...,n).   Then 
there  are  N  =  n. .  m.   cells.  We  have  used  this  partitioning  scheme 
throughout.   Other  sets   S   and  other  partitionings  might  involve 
considerably  more  bookkeeping,  but  the  idea  of  the  method  would  remain 

the  same . 

k  th 

Step  2.   In  order  to  select  the  starting  point  x   for  the  k    mini- 
mization as  far  as  possible  from  the  cells  on  the  list  of  those  which 
have  been  searched,  we  need  a  measure  of  the  distance  d(x,c  )   from  an 
arbitrary  point  x  to  the  center  c   of  the  i    cell  S . .   Two 

distance  functions  were  investigated. 

s      ..,        i.    ,v   n   ,    1x2*1/2  (3) 

a)  d(x,c  )  =  (L=1  (xj"cj^  ' 

the  Euclidean  distance  function,  and 

b)  d(x,c  )  =  max  {|x.-c.|  ,  j=l,...,n  }  (4) 
Then  defining  the  distance  from  x  to  the  territory  already  searched 

to  be  D(x)  =  min  {  d(x,c  )   S.   is  on  the  list  of  cells  already  searched  }  (5) 
we  want  to  choose  x   to  satisfy 

D(xk)  =  max  D(x)  (6) 

xeS 


5. 


that  is,  choose  x   to  be  as  far  as  possible  from  the  closest  cell 


center  which  has  already  been  searched. 

This  could  be  formulated  as  a  deterministic  optimization  problem  — 
for  distance  function  b  a  linear  program  would  result  -  but  it  was 
felt  that  the  effort  of  solving  such  a  program  would  be  excessive. 
Instead,  for  purposes  of  our  algorithms,  the  following  approximate 

heuristic  was  adopted:   Choose  L  points  xeS  at  random,  evaluate 

k 
D(x)   for  each,  and  let  the  next  starting  point  x   be  the  one  of 

these  L  points  with  the  largest  value  of  D(x).   For  our  computations 

L  =  25  was  used  throughout . 

In  the  computational  tests  performed,  no  consistent  superiority  of  either 

distance  function  was  observed.   For  the  algorithms  of  this  paper,  we 

will  concentrate  on  the  Euclidean  distance  function   (3). 

t~Vt 

Step  3.   As  the  k    minimization  proceeds,  the  cells  which  are 
searched  by  this  minimization  must  be  added  to  the  list  to  represent 
the  additional  territory  covered  by  this  search.   For  the  algorithms 
of  this  paper,  a  cell  is  considered  to  have  been  "searched"  if  the 
unconstrained  minimizer  passed  through  it.   In  particular,  most  descent 
methods  for  unconstrained  minimization  involve  a  succession  of  linear 
searches.   For  our  algorithms,  the  cell  which  contains  the  endpoint  of 
each  linear  search  is  added  to  the  list  of  cells  (if  it  is  not  already 
on  the  list).   In  the  early  stages  of  the  algorithm  development,  this 
choice  was  contrasted  with  both  more  and  less  frequent  additions  to  the 
list.   The  methods  were  found  not  to  be  sensitive  to  small  changes  in 
the  frequency  of  addition  to  the  list,  so  this  simple  choice  which 
worked  as  well  as  any  was  selected. 


6. 
Step  4.   The  three  different  algorithms  developed  in  this  paper  differ 
in  the  extent  to  which  the  successive  minimizations  are  completed.   In 
algorithm  Al  ,  each  unconstrained  minimization  is  continued  until  a 
local  minimum  is  found.   In  algorithm  A2  ,  the  k    minimization  is 
terminated  if  it  enters  a  cell  which  has  already  been  found  to  contain 
a  local  minimum  at  some  previous  search.   Algorithm  A3   stops  a  mini- 
mization if  it  enters  a  cell  which  was  searched  (placed  on  list)  by 
any  previous  minimization. 

Algorithms  A2   and  A3  attempt  to  avoid  re-searching  cells  which 
have  already  been  searched,  and  hence  should  be  more  efficient. 

For  purposes  of  comparison,  we  will  also  consider  the  simple 
algorithm  AO  in  which  all  minimizations  are  completed  and  in  which 
the  starting  points  x   are  selected  at  random  in  S.   Thus  for  AO, 

x  =  L.  +  C.  (U.  -  L.)   j=l,...,n 

where  each  £,.      is  a  uniform  random  variable  on  the  interval   [0,1]. 
This  algorithm  makes  no  use  whatever  of  information  from  the  previous 
minimizations . 


IV.   Computational  Tests. 

Computational  experiments  to  compare  the  four  methods  AO,  Al ,  A2, 
and  A3  were  performed  utilizing  the  facilities  of  the  W.  R.  Church 
Computer  Center  at  the  Naval  Postgraduate  School.   For  each  of  the 
test  functions  employed,  each  algorithm  was  run  30  times  with  different 
random  number  sequences;  the  average  of  these  30  runs  is  reported  here. 
A  run  was  allowed  to  continue  until  the  algorithm  had  required  1000 
evaluations  of  the  objective  function   f(x). 

The  same  test  problems  were  used  for  this  study  as  in  Hartman   [2]. 
These  problems,  with  predictable  local  and  global  solutions  were  con- 
structed using  the  objective  function 

f(x)   "  "   l£i  ci  expKx-pjV    k±    (x-pi)] 

This  function  consists  of  the  superposition  of  m  modes,  where  mode  i 
has  depth  c.  e  E  ,  position  p.  e  E  ,  and  shape  and  width  determined 
by  the  n  x  n  negative  definite  matrix  A..  Particular  test  functions 
were  obtained  by  choosing  c.   and  p.   from  a  random  number  table. 

A.   was  then  chosen  to  ensure  that  the  m  modes  were  narrow  enough 
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that  they  did  not  completely  merge  into  one  another. 

Each  of  the  algorithms  requires  an  unconstrained  minimization  method, 
To  establish  comparability  with  an  earlier  study  (Hartman  [2])  Powell's 
derivative  free  method   [3]   was  selected. 


8. 
V.   Results 

The  results  of  the  computational  tests  performed  are  summarized  in 
Tables  1  and  2.   Table  1  gives  characteristics  of  the  10  test  problems 
used.   Table  2  lists,  for  each  problem  and  each  algorithm,  the  average 
(over  30  replications)  best  f  value  obtained  after  250,  500,  750, 
and  1000  function  evaluations.   A  *  indicates  that  all  30  trials 
achieved  the  global  minimum.   The  percentage  of  the  30  trials  which 
did  not  locate  the  global  minimum  after  1000  function  evaluations  is 
also  given  in  Table  2. 

Comparison  of  A0  and  Al   shows  the  advantage  of  using  the 
available  information  in  selecting  starting  points  over  random 
selection  (this  is  the  only  difference  between  A0  and  Al) .   Al 
performed  better  than  A0  on  9  of  the  10  test  functions.   Further 
comparison  of  Al  with  A2  and  A3  shows  the  incremental  advantage 
of  early  termination  of  minimizations  which  are  not  covering  new  ground, 
A3  tends  to  terminate  these  minimizations  more  quickly  than  A2  ,  and 
as  might  be  expected,  it  generally  performs  better.   The  two  variable 
problems  A  -  E  were  handled  very  well  by  A3  ;  all  30  replications 
attained  the  global  minimum  by  the  500th  function  evaluation  for  each 
of  these  functions.   The  five  variable  problems  proved  considerably 
harder  for  all  four  methods,  but  the  methods  which  utilize  past 
information  usually  did  better  than  the  random  method  A0. 

Although  the  algorithms  presented  here  are  generally  successful, 
there  is  clearly  a  need  for  more  work  to  improve  these  methods  and  to 
develop  entirely  new  methods  for  global  optimization. 
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Number  of 

Value  of  Global 

Problem 

Variables 
2 

Minima 
4 

Minimum 

A 

-  9.0 

B 

2 

10 

-  9.9 

C 

2 

10 

-  9.3 

D 

2 

10 

-  9.8 

E 

2 

10 

-13.0 

F 

5 

5 

-  9.4 

G 

5 

5 

-10.1 

H 

5 

10 

-10.0 

I 

5 

10 

-  8.9 

J 

5 

20 

-11.9 

Table  1 


Characteristics  of  Test  Problems 
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Function 


best   f  after  250 
best  f  after  500 
best  f  after  750 
best  f  after  1000 
%  failures 


AO 


Al 


8.7 
8.9 

* 
0.0 


9.0 

* 

* 

0.0 


A2 


* 

* 
0.0 


A3 


* 
* 

* 

0.0 


B 


best  f  after  250 
best  f  after  500 
best  f  after  750 
best  f  after  1000 
%  failures 


9.1 
9.3 
9.8 
9.9 
3.3 


9.3 
9.9 

0.0 


■  9. A 
* 
* 
* 

0.0 


■  9.6 
* 

* 
* 

0.0 


best  f  after  250 
best  f  after  500 
best  f  after  750 
best   f  after  1000 
%  failures 


•  8. A 

■  8.6 

■  8.8 

■  9.0 
26.7 


8.7 
8.9 
9.2 
* 

0.0 


-  9.1 
* 

* 

* 

0.0 


•  9.1 
* 

* 

* 

0.0 


D 


best  f  after  250 
best  f  after  500 
best  f  after  750 
best  f  after  1000 
%  failures 


■  9.3 

■  9.6 
•  9.7 

■  9.8 
10.0 


9.3 
9.8 
9.8 
9.8 
3.3 


■  9.6 
-  9.8 

* 
0.0 


-  9.6 
* 

* 
0.0 


best  f  after  250 
best  f  after  500 
best  f  after  750 
best  f  after  1000 
%  failures 


■10.8 
•12.1 
-12. A 
■12.7 
10.0 


■12.3 

■12.9 

■13.0 

* 

0.0 


■12.6 
■12.8 
■12.9 
■13.0 
3.3 


-12.  A 
■13.0 

* 

* 

0.0 


best  f  after  250 
best  f  after  500 
best   f  after  750 
best  f  after  1000 
%  failures 


■  7.0 

■  8.0 

■  8.6 

■  8.7 
60.0 


-  7.3 

■  8.3 

■  8.6 

■  8.8 
56.7 


■  7.0 

■  8.7 

■  9.0 

■  9.2 
26.6 


■  7.7 

■  8.5 
-  8.9 

■  9.0 
A3. 3 


best  f  after  250 
best   f  after  500 
best   f  after  750 
best  f  after  1000 
%  failures 


■  7.9 

■  9. A 

■  9.8 
-10.1 

3.3 


-  8.9 
■  9.9 
■10.1 
-10.1 
3.3 


-  9.2 
-10.0 
-10.1 
* 

0.0 


-  9.5 
-10.0 

-10.1 
* 

0.0 


*  all  30  trials  attained  global  minimum. 


Table  2 


Test  Results 


Function 


AO 


Al 


A2 


11, 


A3 


H 


best   f  after  250 
best   f   after  500 
best   f   after  750 
best   f   after  1000 
%  failures 


■  7.9  -  8.2  -  8.0  -  7.9 
•  8.4  -  8.7  -  8.7  -  8.7 

■  8.7  -  8.7  -  9.1  -  9.2 

■  9.0  -  9.1  -  9.3  -  9.4 
76.7  70.0  63.3  53.3 


best   f   after  250 
best   f   after  500 
best   f   after  750 
best   f   after  1000 
%  failures 


7.7 

-   8.1 

-    7.7 

-   7.6 

8.0 

-   8.6 

-   8.5 

-   8.4 

8.7 

-   8.7 

-  8.8 

-   8.7 

8.9 

-   8.8 

-  8.8 

-   8.9 

10.0 

16.7 

13.3 

10.0 

best   f   after  250 
best  f  after  500 
best   f   after  750 
best   f   after  1000 
%  failures 


■  7.6  -  6.9  -  7.2  -  7.1 

■  8.8  -  8.0  -  8.1  -  8.0 

■  9.2  -  9.6  -  9.2  -  8.8 
■9.8  -9.9  -9.9  -10.1 

43.3  40.0  43.3  33.3 


*  all  30  trials  attained  global  minimum 


Table  2  (continued) 


Test  Results 
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